Skip to content

Pluggable HPO backends (flaml/hyperopt/optuna) via hiertunehub#342

Open
ZmeiGorynych wants to merge 7 commits into
mainfrom
merge-pr-337
Open

Pluggable HPO backends (flaml/hyperopt/optuna) via hiertunehub#342
ZmeiGorynych wants to merge 7 commits into
mainfrom
merge-pr-337

Conversation

@ZmeiGorynych

Copy link
Copy Markdown
Collaborator

Summary

Ports #337 (pluggable HPO backends via hiertunehub) onto current main (Poetry, econml 0.16, dowhy 0.14), fixes the issues found in that PR, and keeps full parity with today's FLAML options. Original work by @ItsYikunJiang.

CausalTune.fit() now takes framework= ("optuna" default, "hyperopt", "flaml") and algo=, routing estimator-level HPO through hiertunehub.create_tuner instead of flaml.tune.run.

Key decisions

  • Default backend is optuna (MIT, actively maintained, numpy-2 clean). hiertunehub + optuna are hard deps; hyperopt is an optional [hyperopt] extra (it needs setuptools<81 for the removed pkg_resources, so that pin ships only with the extra).
  • Full FLAML-option parity on framework="flaml": cost_attr, low_cost_partial_config, try_init_configs warm-start (points_to_evaluate) and resume (evaluated_rewards, rebuilt from the previous tuner) all preserved. On hyperopt/optuna, resume raises NotImplementedError and try_init_configs warns rather than silently dropping.

Fixes beyond the original PR

  • best_score_by_estimator now takes a list and no longer NameErrors on a malformed entry (referenced a removed loop var).
  • Failure-sentinel sign bug (latent under FLAML, surfaced by the optuna default): a failed trial returned -inf, which a minimizing backend treats as best, poisoning best_estimator. The sentinel is now the worst value for the metric direction (+inf for minimized metrics).
  • model resolves the fitted estimator from scores (the objective pops it out of the result dict in the non-store_all path).
  • Targets pyproject.toml/poetry.lock (the PR edited the now-removed setup.py).

Known upstream (hiertunehub 0.2.1) limitations

  • to_hyperopt() NameErrors when every estimator in the search is parameterless — only affects the optional hyperopt backend with a paramless-only estimator list; optuna/flaml and real multi-estimator searches are unaffected.
  • optuna emits cosmetic UserWarnings for the dict-valued estimator categorical; results are correct.

Tests

New tests/causaltune/test_tuner_backends.py (34 tests): per-framework parse_tuner_params, SearchSpace round-trip, list-based scoring, e2e fits per backend, optuna default + custom sampler, FLAML parity (params reaching flaml.tune.run, resume rebuild), the failed-trial regression, and the best-effort guards. Full non-slow suite: 102 passed, 2 skipped, 0 failed.

DCO

All three commits are signed off by the author (re-ported, not cherry-picked, so no unsigned commits).

🤖 Generated with Claude Code

ZmeiGorynych and others added 7 commits July 8, 2026 13:15
Dependencies for pluggable hyperparameter-optimization backends via
hiertunehub: hiertunehub and optuna as hard deps (optuna is the new
default backend, numpy-2 clean from 4.0), hyperopt as an optional
[hyperopt] extra. hyperopt 0.2.7 imports the deprecated pkg_resources,
removed in setuptools 81, so the setuptools<81 ceiling ships with that
extra. poetry.lock regenerated.

Ports part of PR #337 by Yikun Jiang.

Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Route CausalTune's estimator-level HPO through hiertunehub.create_tuner
so fit() supports flaml, hyperopt and optuna via new framework=/algo=
arguments; optuna is the default. search_space() now returns a
hiertunehub.SearchSpace and parse_tuner_params() maps the tuner settings
to each backend (hyperopt defaults algo to tpe.suggest; optuna/hyperopt
raise on an unbounded search).

Preserve full FLAML-option parity on framework="flaml": cost_attr,
low_cost_partial_config, try_init_configs warm-start (points_to_evaluate)
and resume (evaluated_rewards rebuilt from the previous tuner's results).
On hyperopt/optuna, resume raises NotImplementedError and try_init_configs
warns rather than silently dropping.

Also fix two bugs surfaced by the port:
- best_score_by_estimator iterated a dict and referenced a removed loop
  variable in its error message (NameError); it now takes a list.
- A failed trial returned -inf regardless of metric direction, which a
  minimizing backend (the optuna default) treats as best and so poisons
  best_estimator selection; the failure sentinel is now the worst value
  for the metric direction (+inf for minimized metrics).
  The model property now resolves the fitted estimator from scores, since
  the objective pops it out of the result dict in the non-store_all path.

Ports PR #337 by Yikun Jiang.

Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Cover parse_tuner_params mapping for all three frameworks (incl. the
num_samples==-1 rule, hyperopt tpe.suggest default, exact key sets and
the unbounded-budget guard), search_space returning a hiertunehub
SearchSpace, the list-based best_score_by_estimator (and its ValueError
on a malformed entry), end-to-end fits per backend, the optuna default
and custom sampler, FLAML parity (cost_attr/warm-start/resume params
actually reaching flaml.tune.run, resume rebuilt from the previous
tuner), the failed-trial-not-selected-as-best regression, and the
best-effort NotImplementedError/warn on non-flaml init/resume.

Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
- test_packaging_dependencies: importorskip tomllib (stdlib only from 3.11)
  so the Python 3.10 CI job doesn't ModuleNotFoundError.
- Rename ct.results.trials -> ct.tuner.trials and trial.last_result ->
  trial.result in the experiment runner and example notebooks, following
  the self.results -> self.tuner move. HierTuneHub's Trial exposes .result.

Co-authored-by: Yikun Jiang <jiangyikun9663@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Describe the optuna (default) / hyperopt / flaml backend choice via the
framework and algo arguments to fit(), note that try_init_configs and
resume are flaml-only, and mention the optional [hyperopt] extra. Also
refresh the stale supported-Python line (3.10-3.12) and add optuna to the
dependency lists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
hiertunehub 0.2.1's to_hyperopt() raises a cryptic NameError when the
search space contains only parameterless choices (e.g.
estimator_list=["Dummy"] with outcome_model!="auto"), because it imports
its hyperopt global only while converting a sampler leaf. Raise a clear
ValueError up front instead, pointing at the optuna/flaml alternatives.
Also fix the Mac install guide's stale python=3.9.x.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
test_random_from_biased compares an ERUPT estimate against an empirical
mean with atol=5e-2, using unseeded global np.random. Each CI process
draws different randomness, so it failed intermittently (1 of 6 matrix
jobs). Seed the RNG in make_dataset for deterministic data; seed 0 leaves
~16x margin on the tightest assertion. Unrelated to the HPO-backend change
but surfaced by this PR's CI run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Egor Kraev <egor.kraev@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant